1. Field of the Invention
The invention relates generally to virtual and augmented environments and more specifically to the application of mirror beam-splitters as optical combiners in combination with table-like projection systems that are used to visualize such environments.
2. Background
Virtual Reality (VR) attempts to provide a sense of spatial presence (visual, auditory, or tactile) inside computer-generated synthetic environments to the user. Opaque head-mounted displays (HMDs) and surround-screen (spatially immersive) displays such as CAVEs, (Cruz-Neira, Sandin and DeFanti, 1993) and domed displays (Bennett, 2000) are VR devices that surround the viewer with graphics by filling a great amount of the user""s field of view. To achieve this kind of immersion, however, these devices encapsulate the user from the real world, thus making it in many cases difficult or even impossible to combine them with habitual work environments.
Other, less immersive display technology is more promising to support seamless integration of VR into everyday workplaces. Table-like display devices such as Virtual Tables (Barco Inc., 2000a, 2000b) or Responsive Workbenches (Krxc3xcger and Frxc3x6hlich, 1994; Krxc3xcger, et al., 1995) and wall-like projection systems such as e.g., Powerwalls, (Silicon Graphics, Inc., 1997) allow the user to simultaneously perceive the surrounding real world while working with a virtual environment.
UNC""s xe2x80x9cOffice of the Future Visionxe2x80x9d (Raskar, et al., 1998) is a consequent extension of this concept. Here, in contrast to embedding special display devices into the real work environment, an office is envisioned where the ceiling lights are replaced by cameras and projectors that continuously scan the office environment and project computer graphics to spatially immersive displays that could in effect be almost anything (e.g., walls, tables, cupboards) or anywhere in the office. While the cameras acquire the geometry of the office items (irregular surfaces), the rendering is modified to project graphics onto these surfaces in a way that it looks correct and undistorted to an observer. This concept can offer both, a high degree of immersion and the integration of VR into the habitual workspace.
Due to currently employed display technology, a main drawback of VR is that virtual environments cannot be optically mixed with the real world. If rear-projection systems are employed, real-world objects are always located between the observer and the projection plane, thus occluding the projected graphics and consequently the virtual environment. If front-projection is used, physical models can be augmented with graphics by seamlessly projecting directly onto the surface of those objects instead of displaying them in the viewer""s visual field (Raskar, Welch and Chen, 1999; Raskar, Welch and Fuchs, 1998). However, this so-called Spatially Augmented Reality (SAR) concept is mostly limited to visualization and not suitable for advanced interaction with virtual and augmented real objects. Moreover, shadows that are cast by the physical objects or by the user, and restrictions of the display area (size, shape, and color of the surface) introduce a fundamental problem in SAR systems.
In general, Augmented Reality (AR) superimposes computer-generated graphics onto the user""s view of the real world, thus, in contrast to VR, allowing virtual and real objects to coexist within the same space. Opaque HMDs that display a video-stream of the real world which is premixed with graphics, or see-through HMDs (Sutherland, 1965; Bajura, 1992) that make use of optical combiners (essentially half-silvered mirrors) are currently the two main display devices for AR. Similar to VR, the display technology that is employed for AR introduces a number of drawbacks: For currently available HMDs, display characteristics (e.g., resolution, field-of-view, focal-length, field-of-depth, etc.) and ergonomic factors usually interfere. While the resolution of both HMD types (opaque and see-through) is generally low (lower than projection-based VR display devices), today""s optical see-through systems additionally lack in image brilliance, because the brightness of the displayed graphics strongly depend on the lighting conditions of the surrounding real environment. Although higher-resolution see-through HMDs do exist, e.g. Kaiser Electro-optics, Inc. (2000), they are mostly heavy and expensive, whereas more ergonomic HMDs lack in their optical properties.
Head-mounted projective displays (Parsons and Rolland, 1998; Inami, et al., 2000) or projective head-mounted displays (Kijima and Ojika, 1997) are projection-based alternatives that apply head-mounted miniature projectors instead of miniature displays. Such devices approach to combine the advantages of large projection displays with the ones of head-mounted displays. Similar to SAR, head-mounted projective displays decrease the effect of inconsistency of accommodation and convergence that is related to head-mounted displays. Both, head-mounted projective displays and projective head-mounted displays also address other problems that are related to HMDs: they provide a larger field of view without the application of additional lenses that introduce distorting arbitrations and they prevent incorrect parallax distortions caused by IPD (inter pupil distance) mismatch that occurs if HMDs are worn incorrectly (e.g. if they slip slightly from their designed position). However, as HMDs they seriously suffer from the imbalanced ratio between heavy optics (or projectors) that results in cumbersome and uncomfortable devices or ergonomic devices with a poor image quality.
Although some researchers refer to AR as a variation of VR, e.g. Azuma (1997), a strong separation between AR and VR applications does exist, which, in our opinion, is mainly caused by the technologically constrained usage of different display devices.
In this article, we introduce a prototype of a cost-effective and simple-to-realize optical extension for single-sided or multiple-sided (i.e. L-shaped) table-like projection systems. A large half-silvered mirror beam-splitter is applied to extend both viewing and interaction space beyond the projection boundaries of such devices. The beam-splitter allows a non-simultaneous extension of exclusively virtual environments and enables these VR display devices to support Augmented Reality tasks. Consequently, the presented prototype features a combination of VR and AR. Since table-like display devices can easily be integrated into habitual work-environments, the extension allows the linkage of a virtual with a real work place (e.g., a table-like projection system with a neighboring real workbench).
Compared to current HMDs, the application of a spatial projection displays (such as the prototype described here) for Augmented Reality tasks feature an improved ergonomics, a large field-of-view, a high and scalable resolution, and an easier eye accommodation (Raskar, Welch and Fuchs, 1998). In contrast to Raskar""s SAR concept, however, our optical see-through approach prevents shadow casting and does not restrict the display area to the real environment""s surface.
Since the Extended Virtual Table prototype represents a combination of a table-like display and a mirror beam-splitter this section discusses previous and related works from two areas: table-like projection systems and related mirror displays.
First, we give an overview of current table-like projection technology in subsection 2.1. This is followed by a discussion on related mirror displays in section 2.2.
Table-like Projection Systems
Krxc3xcger""s Responsive Workbench (Krxc3xcger and Frxc3x6hlich, 1994; Krxc3xcger, et al., 1995) is one of the pioneering table-like projection systems. The Responsive Workbench consists of a video projector that projects high-resolution stereoscopic images onto a mirror located under the table, which in turn reflects it in the direction of the table top (a ground glass screen). Analyzing the daily work situation of different types of computer users, Krxc3xcger et al. chose a workbench-like system as an adaptation to the human living and working environment.
Using the Responsive Workbench metaphor, a rich palette of similar rear-projection devices is available today that mainly differ in size, mobility and applied projection technology. Among these systems are Wavefront""s ActiveDesk, Barco""s (2000a) BARON, Fakespace""s Immersadesk Series (Fakespace Systems, Inc., 2000), and also the Responsive Workbench itself, which is sold by TAN Projectiontechnologies (2000).
While all of the above mentioned systems are single-sided projection devices, a few two-sided (L-shaped) systems have been developed to offer a larger and a (by the normally limited projection area) less constrained viewing space. TAN""s Holobench (TAN Projectiontechnologies, 2000), for instance, is an extension of the Responsive Workbench, and Barco""s (2000b) Consul has been developed based on the BARON Virtual Table.
Within the previous six years, an enormous variety of applications (concerning almost all VR areas) that involve table-like projection systems have been described. To mention all of these developments would be beyond the scope of this article.
2.2 Related-Mirror Displays
As for stereoscopic screen-based desktop systems, occlusion caused by the user""s hand or hand-held input devices is a main drawback of table-like rear-projection systems. This disadvantage makes a visually undistorted direct interaction with the presented virtual scene difficultxe2x80x94especially if force-feedback devices such as a PHANTOM (Massie and Salisbury, 1994), etc., are applied to superimpose virtual visual and virtual haptic spaces.
A number of devices have been developed during the last years that allow the user to reach into a virtual scene without causing any occlusion. These, so-called xe2x80x9creach-in systemsxe2x80x9d apply a horizontally arranged small mirror to reflect the graphics that is displayed on a CRT screen (mounted above the mirror). While the user is looking at the mirror, she can simultaneously operate a spatial input device (below the mirror) thatxe2x80x94in most casesxe2x80x94provides force-feedback in relation to the stereoscopically displayed visual information. Since usually neither the input device, nor the user""s hands are visible by looking at the mirror, the virtual environment can be visually perceived in accordance with the corresponding haptic information without causing visual conflicts produced by occlusion.
Knowlton (1977), for instance, overlaid monoscopic 2D keycap graphics on the user""s view of an otherwise conventional keyboard by using a half-silvered mirror that reflected a CRT screen. This allowed the graphics to annotate the user""s fingers (within the illuminated workspace below the mirror) instead of being blocked them.
Schmandt""s Stereoscopic Computer Graphic Workstation (Schmandt, 1983) is another early example of such a reach-in arrangement that applies an electromagnetic tracking device for input in combination with a CRT screen and a half-silvered mirror. He superimposed 3D graphics over the transmitted image of the working area below the mirror.
Poston and Serra (1994) developed the Virtual Workbench, but used a mechanical input device to overcome the magnetic field distortion problems of Schmandt""s setup, which were caused by the interference between the CRT screen and Schmandt""s electromagnetic tracking device.
A more recent development is the apparatus by Wiegand, Schloerb and Sachtler (1999) which they also named Virtual Workbench. Their system offers a trackball for input, a Phantom for input and additional force feedback, and stereo speakers for auditory feedback.
Due to the small working volume of these devices, their applications are limited to near-field operations. Although some of the mentioned systems employ half-silvered mirrors instead of full mirrors for calibration purposes, only a few support Augmented Reality tasks. The maturity of systems, however, renders exclusively virtual xe2x80x98visual and hapticxe2x80x99 information. Several of these devices are commercially available (e.g., the Reach-In Display by Reach-In Technologies (2000) or the Dextroscope by the Medical Imaging Group (2000)) and are mainly used for medical/industrial simulation and training, or psychophysics and training research (Wiegand, Schloerb and Sachtler, 1999).
Bimber, Encarnacxc3xa3o and Schmalstieg (2000a), PCT Patent application PCT/US99/28930, published November 2)) as WO 00/65461, introduced the Transflective Pad, a hand-held half-silvered mirror that was employed in combination with a table-like rear-projection device. The 6DOF (degrees-of-freedom) tracked mirror supported an interactive extension of the limited viewing volume which is provided by such semi-immersive projection devices. It was used, for instance, to view stereoscopically projected volumetric data on a Virtual Table (Wohlfahrter, Encarnacxc3xa3o and Schmalstieg, 2000).
Bimber, Encarnacxc3xa3o and Schmalstieg (2000b), PCT patent application PCT/US99/28930 later extended the concept of the Transflective Pad towards Augmented Reality. In this case, the Transflective Pad was applied as an interactive image plane that folded the viewer""s optical path and merged the reflected graphics with the transmitted image of the real world. Consequently, it represented a possible solution to the occlusion problem that is related to rear-projection systems. The core idea of the Transflective Pad will serve as basis for the optical extension that is described in this article. It is thus an object of the invention to provide improved virtual reality systems.
The object of the invention is attained in the first instance with a virtual environment system that includes apparatus for producing a virtual environment on a projection plane, a planar mirror, and a tracker that tracks the position and orientation of the eyes of a user of the virtual environment system. The planar mirror is of substantial size relative to the projection plane and is positioned relative to the projection plane such that the plane of the mirror intersects the projection plane and the angle of the mirror relative to the projection plane is such that a user of the system who looks at the mirror sees the projection plane reflected. The apparatus for producing the virtual environment receiving a current position of the mirror and producing a first virtual environment on the projection plane when the tracker indicates that the user is looking at the mirror and a second virtual environment on the projection plane when the tracker indicates that the user is looking at the projection plane.
The first virtual environment may be coherent with the second virtual environment or independent of it. In either case, a user may move a virtual object between the first and second virtual environments. When the two virtual environments are coherent, the mirror divides a space having a single global coordinate system which the apparatus for producing the virtual environment employs to produce the virtual environment. In this situation, the apparatus for producing the virtual environment may respond when the user looks at the mirror by producing a first virtual environment which is a view of the portion of the virtual environment which is behind the mirror in the global coordinate system as it would be seen from the direction and point of view of the user if the mirror were transparent and the user were looking through the mirror into the global coordinate system. When the two virtual environments are not coherent, the second virtual environment may function as a magic lens. It may also provide a view of another group that is using a similar virtual environment system.
The mirror may be transflective and there may be a real object which has a location in the global coordinate system on the other side of the system and is visible through the mirror when the object is illuminated. In this situation, the apparatus for producing a virtual environment produces a first virtual environment that augments the real object when the user looks at the mirror. When the virtual reality system is operating in this fashion, it may be used to see how a virtual object interacts with a real object.
Other objects and advantages will be apparent to those skilled in the arts to which the invention pertains upon perusal of the following Detailed Description and drawing, wherein: